Skip to content

Messagebroker feature - #807

Merged
lars-petter-hauge merged 36 commits into
mainfrom
messagebroker-feature
Aug 19, 2026
Merged

Messagebroker feature#807
lars-petter-hauge merged 36 commits into
mainfrom
messagebroker-feature

Conversation

@lars-petter-hauge

@lars-petter-hauge lars-petter-hauge commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Implementing #804

Looks like a lot of changes in this one, but a lot of them are from uv files as each model has their entirely own setup. Also considering that it is a major change in architecture, it isn't as bad as it looks.

git diff --stat origin/main -- . ':(exclude)*uv.lock' ':(exclude)*Docker*' ':(exclude)*docker*' ':(exclude)*radix*'
...
 118 files changed, 4286 insertions(+), 878 deletions(-)

Also excluding test files:

git diff --stat origin/main -- . ':(exclude)*uv.lock' ':(exclude)*Docker*' ':(exclude)*docker*' ':(exclude)*radix*' ':(exclude)*test*'
...
 82 files changed, 2065 insertions(+), 820 deletions(-)

Also, I do not intend to sneak this one in 34951dd

but we haven't required authentication for any models at all. I think we should consider this later if necessary..

A lot of these commits are preparing for change. It is really this commit that changes how the backend works, and which will then rely on a messagebroker in order to function: 7960b7b

Until this part I am actually adding packages to acidwatch backend (that is, those worker projects), and then in this commit removing all those dependencies again. I think this make the git history the most clean and easily readable (fingers crossed..)

Comment thread backend/packages/acidwatch-messaging/src/acidwatch_messaging/transport.py Outdated
Comment thread backend/packages/acidwatch-models/src/acidwatch_models/base.py Outdated
Comment thread backend/src/acidwatch_api/models/__init__.py
Comment thread backend/src/acidwatch_api/database.py Outdated
Comment thread backend/src/acidwatch_api/broker/listener.py
Comment thread backend/src/acidwatch_api/database.py Outdated
Comment thread backend/src/acidwatch_api/routes/models.py
Comment thread backend/tests/broker/integration/conftest.py Outdated
@lars-petter-hauge
lars-petter-hauge marked this pull request as ready for review August 6, 2026 10:46
@lars-petter-hauge
lars-petter-hauge force-pushed the messagebroker-feature branch 3 times, most recently from eec9c9c to 793891c Compare August 7, 2026 13:22
Comment thread backend/src/acidwatch_api/broker/listener.py Outdated
@lars-petter-hauge
lars-petter-hauge force-pushed the messagebroker-feature branch 5 times, most recently from 4c80ff3 to 9810dc6 Compare August 17, 2026 13:36
Move the base adapter and data models into a lightweight package, add
execution-free model definitions, and switch the existing API implementation to
the shared contracts.
Introduce typed adapter jobs, results, heartbeats, and queue naming
Implement durable publishing, acknowledged consumption, result
dead-letter topology, and explicit structural typing checks.

Ensured this works with mypy for the protocols
Provide the message broker communication for each of the adapters
Make it possible to detect if there are workers alive (or if we need to
wait for them to be started)
We add a listener to the results queue who will only ensure that as
messages are coming in from the models, the result is persisted in the
database and if necessary publish the next job.
We start all the backend services from the same location, making it
obvious how many there are.
Utilise the new messagebroker setup to publish simulations from the web
endpoints
Have the models run in-process instead of using web api (all but phpitz)
We ensure that there is a single adapter specification, hence the
workers are not depending on acidwatch_models
Add the heartbeat result in the simulations endpoint response, so that
caller side can know if the simulations are being worked on or not.
@lars-petter-hauge
lars-petter-hauge merged commit 8c150b3 into main Aug 19, 2026
16 checks passed
@lars-petter-hauge
lars-petter-hauge deleted the messagebroker-feature branch August 19, 2026 07:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant